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METHOD OF SEARCHING MULTIMEDIA DATA 

BACKGROUND OF THE INVENTION 
Field Of the Invention 
5 The present invention relates to a method of searching 

multimedia data, and more particularly to a method of searching 
multimedia data more accurately by utilising user feedback* 
Background of the Related Art 

3 Recently; technology for digital image signal processing has 

^0 been greatly developed and has been applied in various fields. 

For example, nhe digital image signal processing technology may 
be used in a search system for automatically editing only the 

- face of a specific character in a moving picture file of a movie 

or drama, in a security system for permitting access to only 

lis persons registered in the system, or in a search system, for 
searching a particular data from an image or video detected by 
a detection system. The performance of such systems basically 
depend on the accuracy and speed of detecting or searching the 
desired object. Accordingly, various image searching methods have 

20 been proposed in the related art. 

An image search system which detects a degree of similarity 
with an, image to be searched utilizing features such as color, 
texture or shape is disclosed in USPAT 5,579,471 entitled "An 
image query system and method*** Depending upon the image to be 

25 searched, the importance of a feature may vary and within one 
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particular feature such as the color, the importance of a feature 
element such as the red or green color may also vary. However, 
the above searching system does not take into consideration the 
different importance of feaT:ures or feature elements for each 
5 image to be searched. 

In another searching method entitled "Virage image search 
engine" (www.virage.com), a user direcLly inputs the level of 
importance for features such as the color, texture and shape by 
O assigning weight values. Although an image may be searched 

J;p according to an importance of a feature using this method, it may 
Ifl be difficult for a user to determine the weights of features. 
In Therefore, Yong Rui in "Relevance feedback techniques in 

interactive" SPIE Vol.3312, discloses a merhod in which images 
U similar to a reference image are found and the importance of 

Mi features or weight for features are automatically obtained by 
Ij calculating the similarities among the found images. However ^ the 
weight importance information is not maintained afrer a search 
for a specific image is finished and must be calculated for each 
image search, even for a same image. 
20 In the image search and browsing system or the video (moving 

image) search and browsing system of the related art, information 
which, describes a particular feature of an image or video data 
is utilized to perform a more effective search or browsing of the 
multimedia data. For example, in the image query system, an image 
25 may be divided into a plurality of regions and a representative 
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color of each region may be utilized as a feature information of 
the image, or a whole color histogram of the image may be 
utilized as a feature information. Thereafter, two images are 
compared zo calculate a similarity based "jpon the feature 
5 information and a determination is made whether the two images 
are similar. 

Therefore, the image search methods in the related art may 
utilize weights of features auch as color/ rexture^ or shape. 
U However, weights of feature elements are not taken into 
18 consideration. Accordingly, the image search methods in, related 
Ifl art has the limitations in intellectually training and developing 
'^S weights for searching, resulting in relatively longer searching 
period to obtain search results and a deterioration of the 
reliability of the search results. 

m 

a SUMMARY OF THE INVENTION 

Accordingly, an object of the present invention is to solve 
at least the problems and disadvantages of the related arr. 

An object of the present invention is to provide a method 
20 of searching multimedia data by automatically updating weighrs 
of features included in a specified image and/or weights of 
feature elements, and by applying the updated weights to search 
for the specified objects 

Another object of the present invention is to provide a 
25 method of searching multimedia data which constructs image 
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characreristi cs corresponding to the zypes of features included 
in a specified image by analyzing and classifying the judgement 
standards applied when the user searches the image^ and adjusts 
the feature information set by taking into consideration weights 
of the features and weights of feature elements during a 
following image search. 

Still another object of the present invention is to provide 
a feature structure to be included in a multimedia data to 
effectively search an image. 

Additional advantages, objects, and features of the 
invention will be set forrh in part in the description which 
follows and in part will become apparent to those having ordinary 
skill in the art upon examination of the following or may be 
learned from practice of the invention. The objects and 
advantages of the invention may be realized and attained as 
particularly pointed out in the appended claims. 

To achieve the objects and in accordance with the purposes 
of the invention, as embodied and broadly described herein, a 
method of searching multimedia data in a multimedia data search 
system comprises searching for a reference multimedia dara 
selected by a user; receiving user feedback of relevance 
information for the searched multimedia data; determining 
importance of respective feature elements of features included 
in the multimedia data according to the relevance information; 
re-performing the search for the reference multimedia data by 

4 
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updating the importance of said respective elements if the user 
requests an additional search; and updating previous importance 
to new importance obtained and maintaining the updated importance 
degrees , 

5 In another embodiment of the present invention, a method of 

searching multimedia data in a multimedia data search system 
comprises receiving an inquiry into previously searched 
multimedia data; analyzing a judgement standard for the 
U multimedia data searched during the inquiry; constructing image 
IS) characteristics using at least one feature included in the 
m multimedia data using an analysis result of the judgement 
standard; and adjusting importance of the image characteristics 
and re-performing a search of the multimedia data if a user 
requests an additional search. 
}^ -^he present invention also provides a feature structure of 

□ multimedia data comprising a first information representing a 
feature of the multimedia data; a second information representing 
a regional feature of the multimedia data; and a third 
information representing importance of the first and second 
20 information. 



BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will be described in detail with reference to 
the following drawings in which like reference numerals refer to 
2 5 like elements wherein: 



5 
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FIG. 1 Shows features in an image by a histogram; 

FIG, 2 shows an image represented by a local grids; 

FIG. 3 is a flowchart of a multimedia data search process 
according to a first embodiment of the present invention; 
5 FIGS* 4 and 5 are flowcharts of a multimedia data search 

process of FIG. 2, where an initial search does not result in a 
desired image; 

FIG. 6 is a basic structure of image characteristics for an 
13 image search according to a second embodiment of rhe present 

±0 invention; 

in 7 is image characteristics of FIG. 6 wherein the 

;^ feature information is constructed using a color and texture; and 

^ FIGS. 8 to 11 are different embodiments of the image 

j-^ characteristics , 

as 

O DETAILED DESCRIP TION OF THE PREFERRED EMBODIMENTS 

Reference will now be made in detail to the preferred 
embodiments of the present invention^ examples of which are 
illustrated in the accompanying drawings. 

^ general image search system, features of an image such 
as color^ texture, and shape may be represented by a histogram 
as shown in FIG. 1. Particularly, FIG. 1 is a color histogram of 
an image by which the colors in the image is grouped into 2 4 
color elements. By adjusting or altering the weights of each 

25 color element, the degree of importance of each color element or 



6 
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rhe extent that a color element affects a search may be 
represented, 

FIG. 2 is an image represented by local grids, Parricularly , 
an image 13 divided into n'^^m grid regions and each grid region 
5 may be represented either by a regional color histogram or by a 
color representing the grid region. Here, the degree pf 
importance of each grid region or zhe extent that a grid region 
affects a search may be represented by assigning weights to each 
O of n^'m grid regions. Also, a predetermined threshold value 
W determined by the system may be utilized as a cap such that 
fli certain grid regions has no influence in the search. Namely, if 
p,; s grid region has a weight which exceeds the predetermined 
U threshold value, the grid region is utilized in the search. 

Otherwise, the grid region is processed as a "Don't care" region 
IE which does not affect the image search. 

|3 image search process will next be explained with 

reference to FIGS, 3 to 5 . 

FIG. 3 is a flowchart of a multimedia data search process 
when a plurality of reference images are selected and weights of 

20 the respective types or elements of features included in a 
specified image are assigned or updated according to the first 
embodiment of the present invention. Referring to FIG. 3, a user 
selects a plurality of reference images (step S301) representing 
the specified image to be searched. Thereafter, the system judges 
25 and determines the importance of the feature elements included 
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in the reference images. 

For example, the system measures the similarities of the 
features included in the reference images (step S302), and 
determines the weights of each feature according to the measured 
similarities of the features (step S303) . Also, the system 
measures the similarities of the feature elements in each feature 
included in rhe reference images (step S304), and determines the 
weights of feature elements in each respective feature according 
to rhe measured similarities of the feature elements (step S3Q5) . 

Accordingly, if a user requests an additional search, the 
system re-performs the image search by using the importance, i.e. 
jj! the weights of zhe features and the weights of the feature 
elements in the respective features (step 3306}. At this time, 
13 the system may use either the weights of the features or the 
3|1 weights of the feature elements, or both. 

k :::: 

rj Particularly, the weights of the features and feature 

'^^^^ elements of respective features are determined as follows. When 
the user selects a plurality of the reference images, the search 
system determines weights of the features and the weights of the 
2 0 feature elements in the respective features by measuring the 
similarities of features of the selected reference image list and 
the similariTzies of the feature elements in the respective 
fea-cures. The weighi:s of the features are calculated by Equations 
la, lb , and Ic below. 



8 
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[la] Weight k = _C<mm_ 

Cont{ALL) 

[lb] Conm = Y, Ya^i<Uj\k) 

m 

[Ic] CortiiALL) = ^ ContCk) 

In the above equations, n denotes the reference number, m 
;=| denotes the number of features used for measuring the similarity, 
Weight_k denotes the weight of the k-th feature, Simx ^k) 
^\ denotes the similarity between the I-th reference image and the 

In j-th reference image when the k-th feature is used, and Cont(k) 

f '1 

. denotes how much the k-th feature contributes to raise the 

|| similarity. Generally, the weights of the respective features 
j: Vle±ght_k increases as the similarities rises since the 
'"f, similarities among the reference images are calculated based upon 
the respective features and the feature having the highest 
similarity acts as the most important factor. 
15 1*he weights of the feature elements are determined by 

Equations 2a and 2b using the similarities of the feature 
elements in the respective features of the reference images. 

[2a] Weight of element i; w^=^^fj{x) 
20 [2b] Similarlity of element i in reference images 



9 
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M0 = ^ 

In rhe above equations, the values of p, and q denote 
constants, m.^ denotes an average of an element i in the reference 
image list, and v^ denotes the distribution of an element i in 
5 the reference image list. According to Equations 2a and 2b, the 
weight of a feature element is inversely proportional to the 
C3 distribution of the corresponding feature element and is 
proportional to an average value of the corresponding feature 
m element component. Therefore, a feature element having a large 

j£0 average value acts as an important factor even if the 

distribution of the corresponding feature element is wide. 
J^f FIG. 4 is a flowchart showing a multimedia data search 

process for when a plurality of reference images are and are not | 
12 selected. Generally, if the number of reference images is not 
15 plural, the image search is performed using features with 

! 

equivalent weights. However, if the search does not result in a 1 

I 

desired image, other reference images similar to the specified ' 
image to be searched are selected and added to a reference image I 
list, Then^ the weights of the respective features and/or feature I 
20 elements are updated using the reference image list. | 
Referring to FIG. 4, when a user selects a plurality of I 
reference images (steps S401 and S402) , the process (steps 
S404-S408) is rhe same as described with reference ro steps 

10 ! 
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3302-5306 of FIG. 3. Thus, an explanation will be omttned. 
Thereafter, if ths user is satisfied with the result of the image 
search (step 3408), the search operation terminates. However, if 
the user is not satisfied with the search result, the user 
5 selects other reference image (s) similar to the specified image 
ro be searched among the resultant images of the search (step 
S411) . On the other hand, when a user selects only one reference 
image, the image search is performed using features with weights 

C3 as assigned (step S403) . Generally, if the reference image is 

XI) selected for the first time, the search is performed using 

ifl features with equal weights assigned. If the user is not 

satisfied with the search result, the user selects other 
reference image (s) similar to the specified image to be searched 

IJ among the resultant images of the search (step S411) , 

Accordingly, the system adds the selected other reference 

|3 image (s) to an initial reference image list managed by the system 

(step S412) . Here, the initial reference image list includes the 
reference image (s) selected in step S401. Thereafter, the system 
measures rhe similarities of the respective features and/or 

20 feature elements in the selected reference images (step S413) , 
and determines the weights of the respective features and/or 
feature elements in the selected reference image by Equations la 
- ICf 2a and 2b (step S414) , Thus, the system updates the weights 
of the respective features and/or feature elements, and re- 

25 performs the image search utilizing the updated weights (step 
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S415) . 

FIG. 5 is a flowchart showing another multimedia data search 
process for when a plurality of reference images are and are not 
selected. Generally, if the number of reference images is not 
5 plural, the image search is performed using features with 
equivalent weights. However, if the search does not result in a 
desired image, other reference images, both similar and 
dissimilar to the specified irr^age to be searched are selected and 
respectively added to a reference image list or a dissimilar 
ii) image list. Then, rhe weights of the respective features and/or 
J^^^ feature elements are updated using rhe reference image list and 

the dissimilar image list. 
13 Referring to FIG. 5, when a user selects a plurality of 

13 reference images {steps S501 and 3502), the process (steps 
y S504-S508) is same as described with reference to steps S302-S306 
p. of FIG, 3, Thus, an explanation will be omitted. Thereafter, if 
the user is satisfied with the result of the image search (step 
S509) , the search operation terminates. However^ if the user is 
not satisfied with the search result, the user selects other 
20 reference image (s) similar and dissimilar to the specified image 
to be searched among the resultant images of the search. On the 
other hand, when a user selects only one reference image, the 
image search is performed using features with weights as assigned 
(step S503)* Here, if the reference image is selected for rhe 
25 first tim,e, the search is performed using features with equal 

12 
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weights assigned. If the user is not satisfied with the search 
result, the user then selects other reference image (s) similar 
and dissimilar to the specified image to be searched among the 
resultant images of the search. 
5 Accordingly, the system adds the selected 3im,ilar image (s) 

to an initial reference image list managed by the system (step 
S511) and adds the dissimilar image (s) to an initial dissimilar 
image list (step S512) , Here^ the initial reference image list 
Q includes the reference image (s) selected in step S501» 
IS Thereafter, the system measures the similarities of the 
If] respective features and/or feature elements in the images 
IZ included in rhe reference image list (step S513), and measures 
"^"^ the similarities of the respective features and/or feature 
elements in the images included in the dissimilar image list 
is {step S514) . 

O Using Equations la - Ic, 2a and 2b, the system, determines 

the weights of the respective features and/or feature elements 
using the images included in the reference image list and the 
dissimilar image list (step S515) . Thus, the system^ updates the 

20 weights of the respective features and/or feature elements, and 
re-performs the image search utilizing the updated weights (step 
S516) , Particularly, the weights of the features by the 
similariry m,easurement of the features and/or feature elements 
in the images included in the reference/dissimilar image lists 

25 are calculated by Equations 3a - 3d. 

13 
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[3a] Wfiight_ k = (aH Weight J ) x (- — ) 

freight^ i 



[ 3b J Weight J, , = (Weight; , ) - ■ ^""^^^^ 



Com (ALL) 



[3d] Coni(ALL) ^^Comik) 

In the above equations^ n denotes the reference number in 
the reference image list or the dissimilar image list, m denotes 
the number of features used for the similarity measurement, 
iJ Wsight_k denotes the final weight of the k-th feature, 
C3 Sxmi (1 ,j ,k) denotes the similarity between the I-th reference 
%§ image and the j-th reference image when the k-th feature is used, 
[3 Cont(k) denotes how much the k-th feature contribute to raise the 
similarity/ Welghtr_i, denotes the weight of the k-th feature in 
the reference image list, and Weightf^^j, denotes the weight of the 
k-th feature in the dissimilar image list. Generally, the 
15 similarities of the images included in the reference image list 
and the dissimilar image list are calculated respectively. As a 
result, the weights of the respective features ^eight_k increase 
as the similarities of the images included in the reference image 
list rises, while the^weights decrease as the similarities in i:he 
20 images included in the dissimilar image list rises. 

14 
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Also, after measuring the similarities of the feature 
elements in the respective features of the images included in the 
reference/dissimilar image list, the weights of the feature 
elements of the respective features are determined by Equations 
-5 4a - 4b, 

[4a] Weight of an element i: Wi=afj{i)'\-hfri{i] 

[4b] Similarity of an element i in the reference images: 

in In the above equations^ f^(I}-pm^xvj^ denotes the 

%U dissimilarity of an element i in the images included in the 
dissimilar image list, the values of a, b/ and q denote 

1^^ constants, /n, denotes an average of the element i in the images 
H included in the corresponding (reference and dissimilar) image 
Vz lists, and v. denotes the distribution of the element i in the 
15 images included in the corresponding {reference and dissimilar) 
image lists. Generally, the similarities of the images included 
in the reference image list and the dissimilar image list are 
calculated respectively. As a result, the weights of the 
respective feature elements Welght^k increase as the similarities 
20 of the images included in the reference image list rises, while 
the weights decrease as the similarities in the images included 
in the dissimilar image list rises. 

If the weights of the features and the weights of the 

15 
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feature elements are determined as above, the similarities will 
be calculated using Equation 5 during the image search. 

n km 

[5] A^Y. ^^-Z ^kjmiF,j,p,q) 

Here, A is a constant, Dlff(F^xPrQ) denotes the difference 
5 between the I-th elements of the k-th feature of the image p and 

image g, w,, , denotes the weight of the I-th feature element of 
12 the k-th feature, Wj, denotes the weight of the k-th feature, n 

denores the number of features, and km denotes the number of 
1% feature elements of the k-th feature* Thu3, the difference is 
is obtained by multiplying the feature difference value of the 
^3 respective image, the feature element weight of rhe respective 
G feature, and the weight of the respective feature. Also, the 
id similarity is obtained by subtracting the difference from the 
□ constant. 

it As described above in reference to FIGS. 3-5, the system 

automatically determines and updates both the feature element 
weights of respective features and the weights of the features 
of the image to be searched when the user searches an image. 
Therefore, a rapid and effective search can be performed, 

20 Nevertheless, if the user wishes to perform a further search 

of the specified image after viewing a previously searched 
result, the user may raise and enter various kinds of queries to 
the search system. Table 1 shows examples of queries by users, 
and Table 2 shows the feature information required according to 

16 
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the type of query when colors and textures are used as the basic 
features of an image. 



Table 1 





Query Type 


1 


What color does image have as a whole? 


2 


Does any portion of the image have a certain color 
feature ? 


3 


About what degree does the portion of the image have a 
certain color feature? 


4 


What texture does the image as a whole? 




r>/*vAe' ^'r-i\» v^v^vt-T^n /^-F -hHe^i nmArrO H<=5\7P r5 f^ertsin tSxtUrS 
UOeS any pon-J-QIl ox une XULa.yt; ticivc; a. ^ w J- i.* ^ + v-i L-4 a- ^ 

feature? 


D 


z!iKr\iTt- uTh;=)t" HArr'rfs^i^ Hrvp^^ thA nortion of the imacje have a 
certain texture feature? 


7 


Does the imaae have a certain color and texture 
feature or have any portion having such features? 


8 


Does any portion of the image have a certain color and 
texture feature? 


9 


About what degree does the portion of the image have a 
certain color and texture feature? 


10 


What color does the image have at a specified 
position? 


11 


What texture does the image have at a specified 
position? 


12 


What color and texture does the image have at a 
specified position? 



17 
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Tabla 2 





Query Typs 


Main Feature Type 


1 


What color does image have as a 
whole? 


information 


2 


Does any portion have a certain 
color feature? 


information 


3 


What degree does the portion 
have a certain color feature? 


j-fOcaj. L.OJ.UZ. 
information 


4 


What texture does the image as a 
whole? 


Global texture 
formation 


5 


Does any portion have a certain 
texrure feature? 


Local texture 
information 


6 


What degree does the portion 
have a certain texture feature? 


Local texture 
information 


7 


Does rhe image have a certain 
color and texture feature or 
have any portiion having such 
features? 


Global color texture 
information 


9 


Doea any portion have a certain 
color and texture feature? 


Local color texture 
1 n r o rma men 


9 


What degree does the portion 
have a certain color and texture 
feature? 


local color texture 
information 


10 


What color does the image have 
at a specified position? 


Local color position 
information 


11 


What texture does the image have 
at a specified position? 


local texture 
position information 


12 


What color and texture does the 
image have at a specified 
position? 


local color texture 
position information 



15 
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In Table 2, 12 query types are presented and ro satisfy the 
characteristics with respect to such queries, the search system 
should have at leasr the following 8 image fearure information. 
The first image feature information is a global color 
5 information which represents the color feature of the whole 
image. A color histogram may be an example of the global color 
information* The second image feature information is a global 
texture information which represents the texture fea-nure of the 
U whole image. A texture histogram may be an example of the global 
IS texture information. The feature information of the color and the 
in texture of the whole image may be represented by a combination 
1% of the global color information and the global texture 
information* 

The third image feature information is a local color 

bfe information which represents the color feature of a region, i.e. 

Ij grid region, in the image. A representative color for each local 

grid may be an example of the local color information. 
Alternatively, the weights of color elements obtained from the 
global color Information may be utilized as the local color 

20 information. 

The fourth image feature element is a local texture 
information representing the texture feature of a grid region in 
the image. A representative texture for each grid may be an 
example of the local texture information. Alternatively, the 

25 weights of texture elements obtained from the global texture 
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information may be utilized as the local texture infomation. 

The fifth image feature element is a local color and texture 
information which represents the color and texture features of 
a grid region in the image. A representative color and texture 
5 for each grid may be an example of the local color and texture 
information. Alternatively, the weights of color and texture 
elements respectively obtained from the global color information 
and the global texture information may be utilized as the local 
""i color and texture information. 

Ho The sixth image feature element is a local color position 

information which represents a color feature in a region at a 

"""^ particular position of the image* A color local grid feature may 

}=f be an example of the local color position information. The 

seventh image feature element is a local texture position 

^is information which represents a texture feature in a region at a 
particular position of the image. A texture local grid feature 
may be an example of the local texture position information* 
Also, the specified color and texture feature in a region at the 
particular position of the image can be represented as a 

2 0 combination of the sixth and seventh information* 

The eighth image feature information is a local color and 
texture information which represents a specified color and 
texture feature in a region at a particular position of the 
image » A color and texture local grid feature may be an example 

25 of the local color and texture position information. 
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The system can perform an effective search by constructing 
a set of feature information^ i.e. image characteristics, as 
described above using the analyzed results based upon the 
contents of the queries and add element weights to the 
5 constructed features. Thus, if a user requests a search, the 
system adjusts the importance of the image characteristics, i,e. 
the weights of the features and/or feature element, and performs 
4f the image search. 

The search method using a reference multimedia data 
%p determines a multimedia data having the highest similarity to the 
reference multimedia data by adjusting the weights of the 
features and/or feature elements of the respective features 
f": included in the multimedia data. Here, the weight adjustment of 
[';; the feature and/or feature elements of the respective features 
% can be performed using one of a direct adjusting method by the 
user, an automatic adjusting method by the system, or an 
adjusting method using the relevance information (i.e., positive 
and negative information) fed back to the system by the user. The 
meanings of the features as described above will now be explained 

20 in detail. 

First, a color histogram represents the color distribution 
in an image. Similarly, the texture histogram represents the 
texture distribution in an image. 

The color image grid represents the color information of a 
25 grid region generated by dividing an image into n^m grid regions. 



The texture image grid represents the texture information of a 
grid region generated by dividing an image into n^m grid regions. 
The color-texture joint local grid represents the color texture 
information of a grid region generated by dividing an image into 
5 n^m grid regions ♦ 

FIG. 6 shows the structure of texture description which can 
be constructed in consideration of the query types and relevance 
tj feedbacks of the user 601. The structure comprises comprises a 
global information 602 which represents a feature of a whole 
m image, a spatial information 603 which represents a feature of 
in an image region, and weight information 604 which represents the 
^= importance of the constructed features 602 and 603. The global 

inform.ation includes a global feature descriptor 605 of the whole 
1^ image, and an element weight descriptor 60 6 of the feature 
Irs elements of the global feature descriptor of the whole image. The 
spatial information 603 includes a spatial feature descriptor 607 
of an image region^ and a position weight descriptor 608 of the 
image region* 

The global information 602 of the whole image and the 
20 spatial information 603 of the image region can be constructed 
by a selective combination of features included in the image such 
as the color, texture, and shape. Here, the possible combinations 
of the basic features can be obtained using Equation 6 below, 
wh^re n denotes the number of the basic features. 
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[6] Y.'^Ck 

Thus^ the number of feature types obtained by Equation 6 
applies to local positions and for global information, since 
there n number of basic features^ the total number of feature 
5 types can be obtained by Equation 7, 

n 

J;J The present invention will be explained utili2ing two basic 

!^[ features of color and texture. In such case, the total number of 
l^" feature types required by the system would be 3H"2*2 = 7, However, 
tt3 if the feature of shape is added, the total number of required 
II feature type would be 7+2*3=13. 

}i FIG, 7 shows image characteristics constructed using the 

features of color and texture. Referring to FIG. 7, the relevance 
feedback image (s) 701 used for adjusting the weights of the image 

15 features according to the user feedback includes global color 
information 702a of the whole image, global texture information 
702b of the whole image, spatial information 703a of image 
regions, spatial color information 703b of image regions, and 
weight descriptor 704 of the global informations 702a and 702b, 

20 and of the spatial informations 703a and 703b. 

In FIG. 7, four feature informations 702a, 702b, 703a and 
7Q3b are used, requiring four weights. Particularly, the global 
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color information 702a includes a global color histogram 705 
representing the color feature information of the whole image, 
and an element weight descriptor 706 of the respective bins of 
the global color histogram. The global texture information 702b 
includes a global texture histogram 707 representing the texture 
information of the whole image, and an element weight descriptor 
708 of the respective bins of the global texture histogram. 

Also, the spatial color information 703a includes a color 
image grid 709, and a position weight descriptor 710 of the color 
image grid. The spatial texture information 7 0 3b includes a 
texture image grid 711, and a position weight descriptor 712 of 
the texture image grid* 

The color histogram 705 is used as a feature information of 
the whole image and the weight of each color element in the color 
histogram 705 are represented by the element weight descriptor 
706. Also, the global texture histogram 707 is used as another 
feature information of the whole image and the weight of each 
texture element in the global texture histogram 707 are 
represented by the element weight descriptor 708. 

Moreover, the color image grid 709 is used as a feature 
information of the image regions and the weight of each grid 
position in the color image grid 709 is represented by the 
position weight descriptor 710- Similarly, the texture image grid 
711 is used as another feature information of the image regions, 
and the weight of each grid position in the texture image grid 
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711 is represented by the position weight descriptor 712. 

As shown in FIG. 7^ an image characteristic structure having 
four feature information was explained in order to satisfy the 
twelve query types in Table 2, However, all nine feature types 
5 for the twelve query types is not necessary. For example, if a 
color-rexture joint local grid is used as a feature, the local 
color, local texture, local color and texture, local ppsition 

Q color^ local position texture and local position color and 
texture can be obtained from the color-texture joint local grid. 

BP Furthermore, in the image characteristic structure of FIG. 

In the feature weights are represented the same level as the 

. feature information, and the feature element weights are 

represented in a level below the respective feature information. 

U: However, image characteristics may be constructed alternatively 

CBS with the feature weights in a level below the respective feature 
information as shown in FIG. 8. For example, assuming that a 
global color information 801 is the feature information, the 
global color information 801 includes a global color feature 802, 
and weights B03. Here the weights 803 is composed of feature 

20 weights 804 corresponding to the global color feature and feature 
element weights 805, 

FIG. 9 shows another embodiment of the image characteristics 
used for adjusting the weights of the image features according 
to the user feedback. In this image characteristic, all 

25 information related to weight characteristics are grouped into 
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a set and represented separately. 

Referring to FIG, 9, the image feature structure 902, i.e. 
the reference feedback, for adjusting the weights of the image 
features when searching the image 901 includes image 
5 characteristics 903 and the weight characteristics 904. The image 
characteristics 903 include global information 905, local 
information 905, and local position information 907* The weight 
characteristics 904 include feature weights 908 and feature 
element weights 909, Moreover, the global information 905 
?;lo includes n feature units 910, the local information 906 includes 
jp: a number of feature units 911 equivalent to a sum of the number 

]^ of features and possible combinations of the features, and the 

local position information 907 also includes n feature units . 

FIG, 10 shows another example of the image data structure 
of FIG. 9 when the image information includes two basic features 
of color and texture. Particularly, the image characteristics 
1001 includes global information 1002, local information 1003 and 
local positional information 1004. The global information 1002 
includes a global color feature unit 1005 and a global texture 
20 feature unit 1007. The local information 1003 includes a local 
color feature unit 1009, a local texture feature unit 1010 and 
a local color and texture feature unit 1011* The local position 
information 1004 includes a local position color feature unit 
1013 and a local position texture feature unit 1014. 
25 Moreover, the global color feature unit 1005 is represented 
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by a global color histogram 1006, the global texture feature unit 
1007 is represented by a global texture histogram 1006, and the 
local color feature unit 1009 and the local position color 
feature unit 1013 are represented by a color image grid 1012. 
5 Also, the local texture feature unit lOlO and the local position 
texture feature unit 1014 are represented by a texture image grid 
1015. Finally, the local color and texture feature unit 1011 is 
represented by both the color image grid 1012 and the texture 
^.3 image grid 1015. 

5:|0 Therefore, the query types in Table 2 can be satisfied by 

constructing image characteristics of the seven features as 
a described above, and the weights are updated by adjusting the 
C3 weights in the feature weights and the feature element weights 
W as shown in FIG. 11. Referring to FIG. 11, the image feature 
lis structure 1102, i.e. the reference feedback, for adjusting the 
weights of the image features when searching the image 1101 
includes image characteristics 1103 and weight charaetsristics 
1104. 

Particularly, the image characteristics 1103 includes global 
20 information 1105, local information 1106 and local positional 
information 1107. The weight characteristics 1104 includes 
feature weights 1108 and feature element weights 1109. Here, the 
global information 1105 includes a global color feature unit 1110 
and a global texture feature unit 1111. The local information 
25 1106 includes a local color feature unit 1112, a local texture 
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teazuxe unit 1113 and a local color and texture feature unit 
1114, The local position information 1107 includes a local 
position color feature unit 1115 and a local position texture 
feature unit 1116, 
5 As described above, according to the present invention, the 

system analyzes all possible queries of the user, and provides 
minimum image characteristics which satisfy all judgement 
standards during the image search. Accordingly, a rapid and 
J; effective image search can be performed by adjusting the weights 
ftfe of the features and feature elements to reflect the user 
in feedbacks. 

7" The foregoing embodiments are merely exemplary and are not 

Tl to be construed as limiting the present invention. The present 
ll teachings can be readily applied to other types of apparatuses. 
SS The description of the present invention is intended to be 
illustrative, and not to limit i:he scope of the claims. Many 
alternatives, modifications, and variations will be apparent to 
those skilled in the art. 
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What is claimed is; 

1. A met:hod of searching multimedia data comprising: 

al) receiving at least one reference multimedia data 
selected by a user, wherein the reference multimedia data 
5 represents a specified multimedia data ::o be searched; and 

bl) searching for the specified multimedia data utilizing 
features and feature elements of the respective features included 
in said at least one reference multimedia data, by considering 
u a degree of affect each feature and each feature elements should 
j| have in the search* 

^3 2, A method of claim. 1, wherein in bl) , searching for the 
O specified multimedia data utilizing a combination of features and 

feature elements of the respective features included in said at 
i| least one reference multimedia data, wherein each feature has a 

feature weight and each feature element has a feature element 

weight , 

3. A method of claim 2, wherein in al) a plurality of reference 
20 multimedia data are selected by the user, and wherein bl) 
comprises ; 

a2) measuring the similarities of features included in the 
pluraliry of reference multimedia data; 

b2) determining weights of each feature according to the 
25 measured similarities of the features in a2); 
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c2) measuring the similarities of ths feature elements in 
each feature included in the plurality of reference multiinedia 
data; 

d2) determining weights of each feature elements in 
5 respective features according to the measured similarities of the 
feature elements in c2>; and 

e2) searching for the specified multimedia data utilising 
C3 features and feature elements included in the plurality of 
reference mulbimedia data, in consideration of the updated 
I^D features weights and feature element weights* 

^ 4. A method of claim 3, further comprising terminating the 

1=- search if the user is satisfied with the result of the search, 
H-^ otherwise the method comprising: 

U fl) receiving at least one other reference multimedia data 

selected from among the resultant images of the search, wherein 
said at least one other reference multimedia data is determined 
to be similar to the specified multimedia data; 

gl) measuring the similarities of features included in the 
20 plurality of reference multimedia data and said at least one 
other reference multimedia data; 

hi) determining and updating weights of each feature 
according to the measured similarities of the features in gl); 
il) measuring the similarities of the feature elements in 
25 each feature included in the plurality of reference multimedia 
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data and said at least one oth^r reference multimedia data; 

jl) determining and updating weights of each feature 
elements in respective features according to the measured 
similarities of the feature elements in il); and 

kl> re-searching for the specified multimedia data utilizing 
features and feature elements included in the plurality of 
reference multimedia data and in said at least one other 
reference multimedia data, in consideration of the updated 
features weights in hi) and feature element weights in jl) . 

5. A method of claim 4, further comprising: 

11) receiving at least one dissimilar multimedia data 
selected from among the resultant images of the search, wherein 
said at least one dissimilar multimedia data is determined to be 
dissimilar to the specified multimedia data; 

ml) measuring the dissimilarities of features included in 
the plurality of reference multimedia data and said at least one 
dissimilar multimedia data; 

nl) measuring the dissimilarities of the feature elements 
in each feature included in the plurality of reference multimedia 
data and said at least one dissimilar multimedia data; and 

wherein in hi) determining and updating weights of each 
feature elements in respective features according to the measured 
dissimilarities of the feature elements in gl and ml); 

wherein in jl) determining and updating weights of each 



feature elements in respective features according to the measured 
dissimilarities of the feature elements in il) and nl) ; and 

wherein in kl) re-searching for the specified multimedia 
data utilizing features and feature elements included in the 
plurality of reference multimedia data, in said at least one 
other reference multimedia data and in said at least one 
dissimilar data, in consideration of the updated features weights 
in hi) and feature element weights in jl) . 

6. A method of claim 3, further comprising terminating the 
search if the user is satisfied with the result of the search, 
otherwise the method comprising: 

f2) receiving at least one dissimilar data multimedia data 
selected from among the resultant images of the search, wherein 
said at least one dissimilar data multimedia data is determined 
to be dissimilar to the specified multimedia data; 

g2) measuring the dissimilarities of features included in 
the plurality of reference multimedia data and said at least one 
dissimilar data multimedia data; 

h2) determining and updating weights of each feature 
according to the measured dissimilarities of the features in g2) ; 

12) measuring the dissimilarities of the feature elements 
in each feature included in the plurality of reference multimedia 
data and said at least one dissimilar data multimedia data; 

j2) determining and updating weights of each feature 
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elements in respective features according to the measured 
dissimilarities of the feature elements in 12) ; and 

k2) re-searching for the specif led multimedia data utilizing 
features and feature elements included in the plurality of 
reference multimedia data and in said at least one dissimilar 
data multimedia data, in consideration of the updated features 
weights in h2) and feature element weights in j2) . 

7, A method of claim 2, wherein in al) one reference multimedia 
data is selected by the user, and wherein b) comprises searching 
for the specified multimedia data utilizing features and feature 
elements of the respective features included in said reference 
multimedia data, \A?herein each features has a feature weight and 
each feature element has a feature element weight, 

8. A method of claim 7, further comprising terminating the 
search if the user is satisfied with the result of the search, 
otherwise the method comprising: 

a3) receiving at least one other reference multimedia data 
selected from among the resultant images of the search, wherein 
said at least one other reference multimedia data is determined 
to be similar to rhe specified multimedia data; 

b3) measuring the similarities of features included in said 
reference multimedia data and said at least one other reference 
multimedia data; 



c3) determining and updating weights of each feature 
according to the measured similarities of the features in b3); 

d3> measuring the similarities of the feature elements in 
each feature included in said reference multimedia data and said 
at least one other reference multimedia data; 

e3) determining and updating weights of each feature 
elements in respective features according to the measured 
similarities of the feature elements in d3) ; and 

f3) re-searching for the specified multimedia data utilizing 
features and feature elements included in said reference 
multimedia data and in said at least one other reference 
multimedia data, in consideration of the updated features weights 
in c3) and feature element weights in e3) . 

9. A method of claim 8, further comprising: 

g3) receiving at least one dissimilar multimedia data 
selected from among the resultant images of the search, wherein 
said at least one dissimilar multimedia data is determined to be 
dissimilar to the specified multimedia data; 

h3) measuring the dissimilarities of features included in 
said reference multimedia data and said at least one dissimilar 
multimedia data; 

13) measuring the dissimilarities of the feature elements 
in each feature included in said reference multimedia data and 
said at least one dissimilar multimedia data; and 



wherein in c3) determining and updating weights of each 
feature elements in respective features according to the measured 
dissimilarities of the feature elements in b3) and h3) ; 

wherein in e3) determining and updating weights of each 
5 feature elements in respective features according to the measured 
dissimilarities of the feature elements in d3) and 13); and 

wherein in f3) re-searching for the specified multimedia 
O data utilising features and feature elem,ents included in said 
reference multimedia data, in said at least one other reference 

19 multimedia data and in said at least one dissimilar data, in 
in consideration of the updated features weights in c3) and feature 

element weights in e3) , 

H 10. A method of claim 7^ further comprising terminating the 
11 search if the user is satisfied with the result of the search, 
otherwise the method comprising: 

a4) receiving at least one dissimilar data multimedia data 
selected from among ^he resultant images of the search; wherein 
said at least one dissimilar data multimedia data is determined 

20 to be dissimilar to the specified multimedia data; 

b4) measuring the dissimilarities of features included in 
said reference multimedia data and said at least one dissimilar 
data multimedia data; 

c4) determining and updating weights of each feature 
25 according to the measured dissimilarities of the features in b4) ; 
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d4) measuring the dissimilarities of the feature elements 
in each feature included in said reference multimedia data and 
said at least one dissimilar data multimedia data; 

e4) determining and updating weights of each feature 
5 elements in respective features according to the measured 
dissimilarities of the feature elements in d4); and 

f4) re-searching for the specified multimedia data utilizing 
features and feature elements included in the plurality of 
^=3 reference multimedia dai:a and in said at" least one dissimilar 
tfb data multimedia data, in consideration of the updated features 
fy weights in c4) and feature element weights in e4). 

C3 11. A method of claim 2, wherein the feature is a color and the 
Ul feature element weights are determined either by a color 
C5 histogram with n color elements as the feature elements, or by 
dividing a multimedia data into n^m grid regions and utilizing 
a regional color histogram or a color representing a grid region 
as the feature elements, 

2 0 12. A method of claim 11, wherein a feature element with a 
feature element weight greater than a specified threshold value 
are utilized in the search. 

13, A data structure comprising: 
25 a feature information including feature and feature elements 
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of an image; and 

a weight infomation including weight information of said 
features and weight information of said feature elements. 

5 14. A method of claim 13, wherein the feature and the feature 
elements are represented by an image characteristic structure 
comprising; 

a global information which represents a feature of a whole 
3 image/ and 

a spatial information which represents a feature of an image 
fij region. 

n 15. A method of claim 14, wherein the image characteristic 
[J structure further comprises a weight information which represents 

hit 

|_j5 the importance of the global information and the spatial 
information, 

16, A method of claim 15, wherein the global information 
includes global feature descriptors of the whole image and 
20 element weight descriptors for the feature elements of the 
respective global feature descriptors, and wherein the spatial 
information includes spatial feature descriptors of image regions 
and position weight descriptors for respective image region, 

25 17. A method of claim 14, wherein the global information 
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includes global feature descriptors of the whole image and a 
weight information which represents the importance of the global 
feature descriptors and the importance of feature elements of the 
respective global feature descriptors; and the spatial 
information includes spatial feature descriptors of image regions 
and a weight information which represents importance of the 
respective image regions 

18. A method of claim 13, wherein the faature and the feature 
elements are represented by an image characteristic structure 
comprising ; 

a global information; 

a local information; and 

a local position information. 

19. A method of claim 18, wherein the features are color and 
texture and wherein the global information includes a global 
color feature unit and a global texture feature unit; the local 
information includes a local color feature unit, a local texture 
feature and a local color and texture feature unit; and the local 
position information includes a local position color feature unit 
and a local position texture feature unit. 

20. A method of claim 19, wherein the global color feature unit 
is represented by a global color histogram, the global texture 
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feature unit is represented by a global texture histogram, the 
local color feature unit and the local position color feature 
unit are represented by a color image grid 1012, the local 
texture feature unit and the local position texture feature unit 
are represented by a texture image grid and the local color and 
texture feature unit is represented by both the color image grid 
and the texture image grid. 
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ABSTRACT OP THE DISCLOSURE 

A m,ethod of searching multimedia data is disclosed in which 
a search for an image can re-performsd by automatically updating 
weights of fearures and/or weights of feature elements in the 
respective feature in an image. 



40 



FIG.l 




123456789 10 11 12 13 14 1516 17 1819 20 21 22 23 24 



FIG. 2 



1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


* « 
































































































59 


60 


61 


62 


63 



FIG.3 



start 



select plural reference images 



i 



measure gimilarities of features 
included in reference images 



S301 



S302 



determine weights of features 
according to measured similarities 
of features 



1 



measure similarities of feature elements 
of features included in reference image 



S303 



S304 



determine weights of feature elements 
of features according to measured 
similarities of feature elements 



S305 



search image by using weights of 
determined features 
and feature elements 



S306 



( end ) 



FIG.4 



( atari } 



select reference image 



S401 



S403 



S402 



■plural reference 
imagesj, 

yes' 



no 



search image by same weights 



measure similarities of features 
included in plural reference images 



determine weights of features 
according to measured 
similarities of features 



S400 



I 



measure similarities of 
elements in features included 
in reference imajges 



T 



determine weights of feature elements 
of features according to measured 
similarities of features elements 



I 



search image by using weights of 

determined features and 
feature elements 



S404 



S405 



S406 



S407 



S408 




lesirec 
iimage exislts in resiii 
listj 

S409 ^ Yes 



select other images similar 
to image to be searched 





add other reference image 
selected to reference list 



S413 



I 



measure similarities of features 
/elements 
included in reference images 



determine weights of 
S414 -^"teature/elements according to 
measured simil arities 

IT 



S415 



, re-pertorm image search by 
updating weights of 
features/elements 



FIG.5 



start ) 

— ^ =t — S501 

select reference image j 



S503 




search image by same weights 



measure similarities of features 
includ ed iii_|]ural reference imagea 



determine weights of features 
according to measured 
similarities of features 



S500 



measure similarities of features elements 
of features includedin reference image -- 



I 



determine weights of feature elements 
of features according to measured 
similarities of feature elements 



I 



search image by using weights of 
determined features and 
feature elements 




S504 



S505 



S506 



S507 



S508 



No 



S5U 



S512 



add other reference 
images similar to ima^e 
.to be searched to reference im age li st 

— 



S513 



S5U 



S515 



S516 



add other reference images ^ ^ 
dissimilar to image to be searched 
to dissimilar image list ^ 

™ 



. measure similarities of features/ 
elements Included in reference imajge list 



measure similarities of features/ 
elements included in diasmilar image list 



I 



determine wekhts of featurea/ 
elements according to measured 
similarities and dissimilarities 



re-perform image search by applying 
determined weights of features/elements 



FIG.6 



601 



relevance 

feedack 
image 



602 

global 
information 



603 



spatial 
information 



604- 



feature 
weights 
descriptor 



605 

global 
feature 
descriptor 



606 607 



feature 
element 

weights 

descriptor 



608 



spatial 
feature 
descriptor 



position 
weights 
descriptor 




O 

2> 





















■a 


1 



FIG. 8 



BOl 

1 

global color 
information 





802 




803 

\ 


global color 
feature 




weights 
characteristics 




805 



feature 
element 
weights 



FIG.9 



image 



901 



903 



image 
characteristics 



905 



contains 
in) 



relevance 
feedback 



902 



904 

4- 



weight 
characteristics 



908 



feature 
weights 



global 
information 



77 



906 



910 



local 
information 





contains 
(n) 



feature 
unit 



911 



contams 

n 

rnCk 
k=l 



feature 
unit 



feature 
element 
weights 



909 



907 

4 



local 
positional 
information 





contains 




in) 




feature 


^912 


unit 





as 
o 



o 











"1 




































■3 






1 


















) 


J 










< 







Docket No.: 



DECLAIIATTQN AKfD POWER OF A^nrORNEY 



I 

As a below numsd inventor, I heceby dednre rfiat: I 

My ceaidencc. post o£nce and dti^en^hip ate as stnted below {lex'f lo my nanie, 

1 believe I ani tlis origmalj first ond sole mvcnrof (if oiiJy onc.naaie is Ubtcd below) oc aa odgmal, fixut and joint invciitoc (if plurai ntuiies 
fire listed below) of die subject ninttcx churned and for whlchjU pfltcjit ib sought on the Lnvciitson CJiudcd 

METHOD OF SEARGHIISIG MULTIMEDI a''!DATA " . the 

spccfficndon o£ which " I 

[X ] ta Eittached hereto [ ] was filed on ^ \ as Applkmon Serial No mid wfis 

amended on. _J f flpplicablc) 

I hereby state that I have reviewed and undersauid the coiue-nts of the above identified speciHcation, iacluding the claiins, as amended by 
my wendii'ient referred ro above, I 

1 acknowledge the duty to disclose mformation wliich is kngwn ro me to be mnrerial to patencabihty m accordance with Tide 37, Code ot 
Federal Regiilations, Section 1,5 6(a). 

I hereby claim iibreigii priority benefics under 35 U.S.C 119(a)[(d) or 365 (b) of miy foreigti applicadQn(a) for patent ormvencor's ccni^cate, 
or 365(a) of any PCT bi^raational applicarion which designated ac least one country odier thm die Umted Scares of Americn, listed below 
and hflve also idendfied below, by cheddng the box, nay foreigti jipplictitioii tor patent or inventor's cerdHcate. or oC any PCT intcmational 
application having a £.ling date before that oE the appliciitioni [on which pdority \^ claimed 



Prior Foreign Applicadon(8): 




Foreign Filing Date 


Number 


Country 


Month/Dav/Vcar 


3162/1999 


Korea 


February/1/1999 


3183/1999 


Korea 


f$brudrY/l/l999 







< 



I hereby claim tiie benefit under 35 U.SC U9{e) of any Unijred Stiires provisional appUcaaon(s) hsmd below. 
Application Numberra^: Biling Date nvlonth/DWYfiar) 



I hereby cJaim the benefitundex 35 U.S-C, 120 oftiny United g^iates applicat3on(s), or 365(c) of any PCT incemadonal fippiicimon designarine 
die United States of Amedca, liatcd below and, insofar as tinp|3ubject cnuLter oc each ofdie claims ot thia appJicadon I3 not disclosed in the 
prior United States or PCf international applicndou in the ma|u\er provided by the first paragraph of 35 U.S.C. 112, 1 acknowledge the dut) 
to disclose tnfon-natioA which ia mfitedfil to patentabiiity ti$ d^Si'ied u\ 37 CFR 1.56 which became fivallsble between the filing date of tlif 
pdor npplicadon and the ntitioniil or PCI" intemfttlonal filing', dale of dii? Hpplicadon- 

Prior U, S. Application 

or PCT Parent Number Filing Date (Monch/Pay/Yegrl Parent Patent Number (if apoUcnble 

i 



I hereby declare diat all statements made herein of my own jtnowledge are tme iifid tliat all scatenienrs made on infomiation and belief ar: 
l^elieved to be li-ue; and furriier diat d^ese sta-tenients svi;re rti'ade wi(h die kt^owledge ih^t willful f>\lse 3talen\ent3 and the like so m^de a.ff 
punishable by fine or imprisonment, or both, andec Section IpOl ofTide 1 8 of the United States Code and that such willful false sratemenc 
m?iy jeopardise the valLdity of die application oc ?iny patent ijrsued thereon. 

1 

( 

i 

! 



T hereby .ippoiiu rhe following ii«QC3iey(i;) :Hid/or iigeiit(s):Dri;mcl YJ. r-Gni, Regis tration No. 36^186 aad Mnrk L Plcjihncr, RegihtfflHon No 
34,596; CudJL Wesolowsld. RKgiaicarionK'o. 40,372Jolin C Ei^cnhnrii, Rep^stcntioaNo. 38,1 Kcnc A. Vaiique^, Regis reran on No. 38.647 
MicWnslJ. Corncliso3i,Registi::idonNo. -^10,395; and ScuiittLSn'ilth,Rcgi:^tnidon No. 42jl59iaiid Cfirol U Dmzbick, Registration No, 40,287 
dl of. 

FLEST-INER 6c 1<XU 

?, O. Box 221200 

ChaatOiy, Yirgi'nin 20153-1.200 

Wi&L full [jowei: otiub Slim don and revocation, to pcosecute tl^is ;ipp]tcadon mc\ fo tcuiisacr all business in tlie Pacent and Trademark Officf 
coniiecLed dierewirh. and nil futivre correspondence should be addressed co diem. 

^1, f 1 ^ ' Jin Soo LES 

Full nniTie or ?o]c oc fitrst inverijpi:: 



Reiitdcnce; 




Inventor's sip,nat:ute; , DataJanyayy ,29 , 200 0 

Seo^27, Korea 



Cmzenship: Republic of Korea 



Post Office Address: ^^^^^ 101^809/ J.36, Koyo 1-dong/ Songpa-gu / Seoul ^ Korea 

Hyeon Jun KIW 



Full name of jomt invenxorfs): 



Inventor's siffliaturc: ^"^^ ■ Date January '29 y 200 0 

Residence: Kyonggi-do , Korea 



Republic of Korea 
Cm2ensh:p: 



Po5t Qrfice AddccastHanshin life 1 09-302 /Punq dang -dona f Pundancf-Qu / Kyonqqi -do / Korea 



FuU nmne of joint mventot(<) : 



In yen cot's si e:n re : 



Residence* Date 



Cidzen<ih\p: 



Po3r Office Addressi 



Full nomc of joint inveator(3): ^ ^ 

Inveatoc'a siffliatutc : 

Residence: Dare : 



Cii:i;;enshjp: 



Post Office Address:, 



